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This  publication  is  primarily  a  working  paper.  It  is  published  solely  to  document  work  performed. 


SUMMARY 


This  paper  describes  a  prototype  computer  software  system  automating  the  design  of  computer 
displays.  The  information  displayed  includes  tables  (e.g.,  as  parts  tables  or  diagnostic  tables)  and 
functional  diagrams. 

The  concept  of  operations  for  the  Integrated  Maintenance  Information  System  (IMIS)  requires 
that  all  technical  data  be  stored  in  a  database  independent  of  screen  formatting  information.  For 
example,  the  electronic  form  includes  no  information  about  where  a  box  of  the  diagram  is  to  be 
placed,  or  even  that  some  information  is  to  be  displayed  as  a  diagram  instead  of  a  table.  Hence, 
this  format-neutral  storage  representation  then  allows  software  to  automatically  reformat  the  data 
for  new  computer  screen  sizes,  new  portables,  new  color  capabilities,  or  new  methods  including 
animation  or  head-mounted  displays.  This  separation  of  information  content  from  formatting  also 
permits  better  analysis  to  check  for  consistency  or  to  merge  information  from  multiple  sources. 
When  there  are  changes,  minimal  human  reengineering  effort  is  required  to  update  the  information 
and  the  displays,  resulting  in  reduced  costs  and  higher  quality. 

The  automatic  design  and  layout  methods  presented  here  represent  a  new  capability.  Previous 
layout  techniques  exist  but  have  been  limited  to  a  few  preselected  screen  structures  whose  rendering 
does  not  adapt  to  new  data.  In  addition,  these  previous  methods  do  not  adapt  to  algorithmic 
techniques  integrated  with  human  factor  constraints  to  select  high-quality,  dependable  displays. 

Our  approach  may  be  termed  rule-based. ,  since  each  table  or  diagram  is  automatically  generated 
from  a  set  of  rules  dictating  what  constitutes  an  acceptable  rendering  on  the  screen.  The  rule-based 
approach  makes  it  easy  to  incorporate  human  factors  rules  for  guiding  the  selection  of  appropriate 
displays. 

The  capability  to  build  a  computer  system  to  carry  out  these  logic-based  designs  has  been 
enabled  by  advances  in  theory  and  in  technology  including  very-high-level  languages,  more  advanced 
compilers,  and  representation  transformation  tools. 

The  main  principle  used  by  the  rule-based  system  is  that  items  associated  in  the  database, 
such  as  a  symptom-fault  pair,  must  be  given  a  visual  association  on  the  screen,  such  as  occurring 
in  the  same  row  of  a  table.  Elaborations  of  this  and  related  principles  are  presented  and  illustrated 
in  this  paper. 

Our  rule-based  prototype  has  been  demonstrated  with  maintenance  technical  data  provided  by 
the  Air  Force  Human  Resources  Laboratory  (AFHRL).  This  paper  presents  a  series  of  tables  and 
functional  diagrams  automatically  designed  by  the  prototype  computer  software  system.  The  tables 
contain  fault,  symptom,  and  test  data,  and  the  functional  diagrams  are  for  an  aircraft  subsystem. 
Paper  (not  electronic)  versions  are  also  produced  automatically  (the  tables  and  diagrams  in  this 
paper  were  produced  this  way). 
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PREFACE 


The  work  described  in  this  technical  paper  was  performed  by  the  Kestrel  Institute  under 
subcontract  from  Systems  Exploration,  Inc.  (SEI).  The  work  was  accomplished  for  the  Air  Force 
Human  Resources  Laboratory  under  contract  F33615-88-C-0004,  task  number  10.  The  work  is  in 
support  of  AFHRL’s  Project  2950,  Integrated  Maintenance  Information  System.  Mr.  David  R. 
Gunning  monitored  the  contract  for  AFHRL. 

This  paper  is  concerned  with  the  automated  display  of  neutral  maintenance  data.  The  display 
technology  described  in  this  paper  establishes  the  feasibility  of  this  approach  for  the  IMIS  and 
provides  an  initial  prototype  for  evaluation. 

The  prototype  is  written  in  the  Refine™  language,  running  on  Sun  4  workstations.  The 
drawing  routines  and  the  user  interface  were  implemented  within  the  X-ll  window  system  running 
under  Intervista™. 

The  original  basis  of  this  work  came  from  an  idea  suggested  by  Cordell  Green  and  elaborated 
into  a  suggestive  set  of  rules  by  David  Gunning  for  a  class  project  at  Stanford  University.  We  would 
also  like  to  acknowledge  the  guidance  of  David  Gunning  in  (a)  providing  the  initial  theoretical 
foundation  for  our  logic-based  approach  while  he  was  a  graduate  student  at  Stanford,  and  (b) 
showing  how  our  research  could  be  applied  to  the  IMIS  technical  data  design  problem. 

The  transformational  theory  for  tables  developed  and  presented  herein  is  from  work  by  Stephen 
Westfold,  Cordell  Green  and  Janet  Coursey.  Stephen  Westfold  and  Janet  Coursey  implemented  the 
table  generator.  David  Zimmerman  and  Stephen  Westfold  designed  and  implemented  the  diagram 
generator.  David  Zimmerman  and  Li-mei  Gilham  implemented  the  user  interface.  Westfold  and 
Zimmerman  put  together  all  the  pieces  to  build  the  integrated  prototype  system. 

We  would  like  to  acknowledge  Mark  Miller  of  SEI  in  helping  us  through  the  myriad  of  technical 
and  administrative  obstacles  we  encountered  doing  this  work.  We  acknowledge  the  valuable  time 
spent  by  Li-Mei  Gilham  developing  our  user  interface.  We  acknowledge  Jorge  Phillips'  efforts 
coding  the  box-rendering  routines,  Marilyn  Daum’s  design  and  implementation  of  a  menu  system, 
and  Dick  King’s  help  with  SGML  and  authoring.  We  also  acknowledge  assistance  from  Lt.  Janet 
Murphy  of  AFHRL,  and  Doug  Andrews  and  Ross  Rannels  of  Systems  Research  Laboratories. 
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I.  INTRODUCTION 


Recent  advances  in  technology  for  greater  automation  in  software  include  knowledge- based 
methods  for  software  creation  and  logical  specification  of  software  using  very-high-level  specifica¬ 
tions  (Smith,  1988),  (Kotik,  Rockmore,  &  Smith  1986).  This  advanced  technology  can  be  applied 
to  the  automated  generation  of  graphical  user  interface  software.  The  key  observation  is  display 
synthesis  is  analogous  to  software  synthesis. 

We  illustrate  herein  a  theory  for  the  creation  of  tables  and  diagrams.  We  plan  to  achieve 
for  automated  display  synthesis  the  large  increase  in  productivity  and  maintainability  automated 
programming  brings  to  software  production  and  maintenance.  Display  or  presentat  ion  specifications 
are  software,  too  and  are  subject  to  modification  and  in  need  of  maintenance.  Users  want  fast  and 
inexpensive  turnaround  to  test  requirements  before  making  a  larger  investment. 

Such  automated  design  of  graphical  representations  can  (a)  increase  productivity,  (b)  allow 
the  rapid  exploration  of  a  large  number  of  alternative  display  designs,  and  (c)  be  used  to  produce 
standard  display  formats  from  unformatted  data.  It  allows  for  the  generation  of  displays  for  unan¬ 
ticipated  cases  (i.e.,  a  rule-based  design  system  effectively  determines  what  kind  of  displays  can  be 
effective  in  communicating  new  information). 

The  significance  of  this  technology  for  the  Integrated  Maintenance  Information  System  (IMIS) 
project  is  severalfold.  First,  the  IMIS  will  apply  the  concept  of  storing  format-neutral  data  in  the 
repository  of  technical  data.  The  data  are  neutral  with  respect  to  display  and  formatting.  Thus, 
as  new  display  consoles,  new  maintenance  workstations,  new  screen  look-and-feel  standards,  and 
new  printing  standards  emerge,  the  data  can  automatically  be  formatted  to  comply  with  these  new 
constraints.  By  driving  all  the  display  and  print  systems  off  the  same  neutral  data,  consistency  is 
established  across  the  various  media.  As  new  techniques  such  as  color,  animation  or  virtual  reality 
displays  are  introduced,  little  or  no  change  to  the  technical  data  will  be  required.  The  availability  of 
a  low-cost  display  design  capability  allows  the  technical  data  to  be  freed  from  formatting  information 
so  inference  systems  can  analyze  the  data  for  consistency,  quality,  optimization,  upgrades,  and  so  on. 
Also,  technical  data  from  multiple  vendors  can  be  merged  into  combined  maintenance  presentations 
using  data  structure  and  procedure  reformulation  techniques. 

This  work  differs  from  the  conventional  notion  of  a  "screen  generator. ’’  The  generative  theory 
adapts  to  new  types  of  data  or  display  criteria  because  it  uses  the  structure  and  contents  of  the 
data  t-j  guide  the  creation  of  the  display.  We  illustrate  these  capabilities  in  detail  in  this  paper. 

Two  main  categories  of  choice  are  provided  by  the  synthesizer:  (a)  data  structure  reformulation, 
and  (b)  variation  of  display  styles.  The  data  reformulation  steps  reorganize  the  information  directly 
so  the  system  can  synthesize  a  greater  variety  of  displays  than  is  possible  by  a  system  that  fits  the 
data  within  a  predefined  display  format. 

There  are  subsystems  for  generating  tables  and  diagrams.  These  two  subsystems  share  the  same 
graphics  bounding-box  model  and  implementation  as  well  as  many  of  the  same  design  principles. 
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Organization  of  the  Report 


Section  II  describes  our  technical  approach.  Section  III  describes  the  table  generator  in  more 
detail  and  presents  an  annotated  trace  as  the  prototype  designs  a  series  <  f  tables.  Section  IV 
describes  the  diagram  synthesis  prototype.  Again,  several  examples  of  generated  displays  help 
illustrate  the  process.  Section  V  suggests  several  future  directions  and  contrasts  this  research  with 
some  related  work  in  the  area.  The  last  section  offers  our  conclusions  regarding  this  first  phase  of 
work. 


II.  TECHNICAL  APPROACH 

Generator/Selector  Factoring  of  Theory 


We  are  interested  in  automated  design  assistance  for  the  generation  of  graphical  user-interface 
software.  The  research  strategy  chosen  is  to  divide  the  problem  into  a  generator  theory  and  a 
selector  theory. 

A  generative  theory  takes  as  input  some  information  to  be  displayed  graphically  and  generates 
as  output,  all  plausible  graphical  ways  to  display  that  information.  Such  a  generative  theory  forms 
the  basis  for  the  automatic  synthesis  of  graphical  displays.  The  degree  to  which  the  theory  can 
generate  all  plausible  displays  determines  the  theory’s  flexibility  and  utility. 

A  selector  theory  takes  as  input  a  set  of  displays  and  places  as  output  the  displays  in  order  of 
preference  (or  just  names  one  that  is  satisfactory).  The  selector  theory  consists  of  a  set  of  criteria  by 
which  the  displays  are  filtered  so  preferred  displays  are  separated  from  the  many  possible  displays 
generated.  In  the  implementation,  the  selector  prunes  the  space  of  possible  displays  proposed  by 
the  generator. 

We  are  following  the  research  strategy  of  first  pursuing  the  development  of  the  generative  theory 
and  later  combining  it  with  a  selector  theory.  Accordingly,  in  this  research  effort  we  have  taken 
the  important  first  step  and  developed  the  generative  theory  for  the  automated  synthesis  of  tables 
and  diagrams. 

Human  factors  determine  the  selector  theory.  For  example,  the  applicable  human  factor  analysis 
may  call  for  limits  on  the  ratio  of  line  width  to  character  size,  or  on  the  number  of  functional  blocks 
shown  in  a  limited  space.  Such  criteria  would  be  stated  in  the  selector  theory. 

This  division  of  the  theory  into  a  generator  and  a  selector  is  used  for  theory  development 
and  explanatory  purposes.  Thus,  the  concept  is  simply  that  the  generator  produces  the  space  of 
presentations  and  the  selector  prunes  this  space.  However,  for  efficiency,  the  implementation  can  be 
made  more  efficient  by  incorporating  the  se  ;ior  rules  into  the  generator  (enabling  rapid  pruning 
of  the  design  tree). 

To  limit  the  scope  of  this  initial  investigation  into  the  theory,  we  have  restricted  the  type 
of  graphical  displays  generated  to  forms  of  tables  and  diagrams  (graphs).  It  does  not  include 
animation,  color,  or  solid  modeling,  although  we  see  no  intrinsic  limitations  on  extending  the 
methodology  into  these  areas. 
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Analogy  to  Program  Synthesis 


Graphical  screen  synthesis  is  quite  analogous  to  conventional  program  synthesis.  In  particular 
the  two  notions  of  data  structure  reformulation  and  program  optimization  are  directly  applicable 
to  graphics  synthesis.  For  example,  the  common  program  optimization  that  consists  of  pulling 
constants  out  of  a  loop  has  a  direct  analogue  in  graphics,  such  as  removing  constant  expressions 
from  visual  representations  of  sets  or  sequences  and  only  presenting  the  constant  expression  once 
(e.g.,  as  a  heading  for  a  column  of  elements). 

In  addition  to  these  particular  principles  that  carry  over  directly,  the  larger  methodology  of 
program  synthesis  also  fits  graphics  synthesis.  One  example  is  the  use  of  our  technology  of  transfor¬ 
mations  to  transform  internal  information  structures  into  external  screen  entities.  Another  example, 
analogous  to  the  display  generator  and  selector  discussed  above  and  found  in  conventional  program 
synthesis,  is  an  efficiency  estimator  (selector)  needed  to  guide  the  heuristic  search  for  an  efficient 
implementation  since  all  possible  implementations  (produced  by  the  generator)  cannot  be  searched. 
Graphics  synthesis  also  needs  an  efficiency  estimator — one  that  estimates  how  well  the  ultimate 
execution  engine  of  the  graphics,  namely  the  human  user,  will  be  able  to  “execute5’  or  use  the 
graphics  structure  to  solve  the  problem. 


Progress 


We  have  developed  the  rule  base  necessary  to  synthesize  programs  that  design  displays  of 
relations  as  diagrams  and  tables.  We  have  been  able  to  test  this  theory  by  implementing  a  simple 
prototype  graphics  generator  proving  the  concept. 

The  problem  has  proved  to  be  tractable  with  our  methodology  and  we  are  optimistic  about  the 
future  possibilities.  The  theory  has  fulfilled  our  hopes  in  several  ways:  (a)  the  theory  did  indeed 
turn  out  to  be  simple  (after  we  had  worked  it  out),  (b)  we  were  able  to  build  upon  our  transforma¬ 
tional  foundation  and  experience  in  automatic  programming  to  gain  considerable  leverage,  (c)  the 
implementation  of  a  prototype  was  straightforward,  consisting  of  expressing  the  graphics  theory  as 
a  set  of  compilable  transformation  rules,  and  (d)  the  theory  can  be  immedirtely  extended  in  several 
directions,  as  discussed  in  the  conclusion  of  this  paper. 

We  have  applied  our  prototype  display  generator  to  several  forms  of  input  including  sets, 
mappings  and  relations.  It  produced  many  different  types  of  tables  and  diagrams.  The  example  in 
Section  III  shows  a  series  of  tables  automatically  generated  from  a  binary  relation.  The  examples  in 
Section  IV  show  diagram  layouts  automatically  generated  from  just  the  connectivity  information. 

Since  paper  “displays”  are  also  a  target  of  this  work,  we  have  implemented  a  preliminary  set 
of  rules  for  producing  printable  representations  (as  TgX  files)  of  generated  displays.  We  have  also 
constructed  a  menu-based  irterface  to  the  generator,  which  allows  the  user  to  select  various  options, 
enter  new  input,  and  cycle  through  a  generation  sequence.  A  screen  dump  of  this  interface  is  shown 
in  Figure  1.  All  of  the  diagrams  ami  tables  reproduced  in  this  paper  were  generated  automatically 
by  the  sys‘em  and  used  as  is. 


Figure  1.  Menu- Based  Interface  to  Display  Generator 


Guiding  Principles 

This  theory  was  inspired  by  the  observation  that  it  seems  possible  to  generate  displays  using 
only  a  few  simple  principles.  One  idea  is  to  show  that  two  items  are  associated  to  show  a  simple 
visual  association.  That  is.  internal  commonality  can  be  communicated  by  visual  commonality.  For 
example,  if  two  objects  in  a  database  share  a  common  property,  such  as  occurring  in  the  same  field 
of  a  relation,  then  that  association  ran  be  shown  on  the  screen  by  having  tin  tv,,  objects  share  a 
common  screen  property  such  as  horizontal  or  vertical  position. 

One  step  further  than  this  simple  pairwise  association  is  the  notion  of  ordering.  An  ordering 
in  the  data  can  be  communicated  to  the  external  world  by  employing  some  sort  of  visual  ordering, 
such  as  increasing  the  y  position,  intensity,  or  angle. 

Two  basic  principles  guide  the  synthesis  of  the  display:  (a)  visual  encoding  of  the  semantic 
role  of  the  data  structure’s  parts,  and  (b)  optimization.  The  contents  of  the  data  belong  to  the 
same  structure  because  they  share  some  common  property  or  have  a  certain  relationship  with  each 


other.  The  visual  display  should  convey  this  information. 

The  display  should  strike  a  balance  between  useful  redundancy  and  unnecessary  clutter.  Opti¬ 
mization  helps  remove  undesirable  redundant  parts  of  the  presentation,  and  make  the  best  use  of 
the  selected  visual  display  properties. 

1.  Semantic  and  visual  similarity.  A  shared  property  of  the  items  being  displayed  is  shown 
as  some  shared  visual  property  on  the  screen — vertical  alignment,  horizontal  alignment,  being  joined 
by  arcs,  or  table  position.  Prettyprinting  is  an  example  in  which  the  items  at  the  same  depth  in 
the  expression  tree  share  the  visual  property  of  the  same  level  of  indentation. 

The  semantic  properties  of  the  data  structures  in  our  examples  include  membership  in  a  set 
or  sequence,  and  fields  of  the  same  tuple.  A  subclass  of  sameness  is  association  between  items,  as 
defined  by  a  mapping  or  a  relation.  It  is  reflected  as  a  visual  association:  (a)  horizontal  or  vertical 
juxtaposition,  (b)  “labels”  or,  (c)  line  segments  joining  the  items. 

2.  Optimization.  When  displaying  complex  data,  or  data  with  labels,  parts  of  the  relation 
or  the  labels  may  appear  multiple  times  because  they  are  associated  with  each  datum.  A  common 
subexpression  removal  type  optimization  can  eliminate  some  of  the  repeated  patterns.  This  op¬ 
timization  is  analogous  to  standard  compiler  optimization  techniques,  (such  as  finite  differencing, 
pulling  constant  calculations  out  of  loops,  loop  combining,  etc.,)  that  identify  and  remove  repeated 
computations  from  programs. 

3.  Well-ordering.  Often  the  information  to  be  displayed  is  ordered  (alphabetically,  numeri¬ 
cally,  by  region,  and  so  on).  Even  if  the  ordering  is  not  explicitly  present  in  the  data,  it  is  sometimes 
useful  to  choose  an  ordering  principle  to  help  structure  the  display.  The  intended  use  of  the  data 
will  govern  the  ordering  for  example,  telephone  directories  ordered  alphabetically  by  surname  or 
by  street  address.  A  radar  display  uses  a  radial  ordering  to  sweep  out  the  field  of  view. 


Our  goal  is  to  develop  a  robust  generator.  This  effort  requires  a  very  fine-grained  theory:  the 
ability  to  generate  all  the  plausible  variations  of  graphic  displays  from  all  plausible  data.  The  fine 
granularity  of  the  reformulation  steps  and  the  details  within  each  display  style  is  what  guarantees 
adequate  coverage  of  the  space  of  possible  graphic  representations.  Restated,  we  want  the  grain 
size  of  the  display  decisions  (such  as  whether  to  center  or  justify  text,  font  choice,  and  so  on) 
to  be  fine  enough  to  generate  a  wide  spectrum  of  displays.  Then  some  of  the  displays  are  likely 
to  be  particularly  effective  for  communicating  a  given  set  of  information.  We  can  characterize 
such  a  theory  as  robust,  not  brittle.  In  contrast,  a  generator  that  selects  displays  from  several 
large,  precanned  display  types  cannot  be  gracefully  extended.  By  allowing  subtle  variations  to  be 
generated,  we  have  the  flexibility  to  handle  new  data  and  new  types  of  display  media. 


Simple  Examples 

We  now  look  at  two  displays  for  small  data  structures  to  illustrate  the  application  of  the 
association  and  optimization  principles. 

An  element  in  a  complex  data  structure  plays  one  or  more  roles  in  relation  to  the  whole.  For 
example,  the  elements  of  a  set  are  in  the  membership  role  with  respect  to  the  whole.  Corresponding 


domain  and  range  elements  from  a  map  are  in  one  role  of  associating  one  with  the  other,  and  another 
role  of  being  one  entry  from  a  larger  map.  In  displaying  a  data  structure,  we  choose  some  of  these 
roles  to  manifest  visually.  Some  visual  trait  will  be  selected  to  indicate  each  chosen  role.  The 
possible  choices  for  the  visual  sameness  are  same-row,  same-column,  same-font,  arc-plot  (tables 
with  line  segments  joining  related  entries),  and  offset-plot  (tables  indexed  by  row  and  column 
position). 


Tuple 

For  a  pair  (a,  b),  the  property  shared  by  a  and  b  is  membership  in  the  same  tuple;  that  property 
could  be  visually  represented  as  either  vertical  or  horizontal  alignment  (same-column  or  same-row): 

1  <1 
a  b  or 

b 


Sequence 


The  increasing  order  of  the  elements  in  a  sequence  can  be  shown  by  incrementing  some  visual 
trait  (e.g.,  vertical  or  horizontal  position,  intensity,  size,  and  so  on)  as  each  element  is  displayed. 
That  the  elements  belong  to  the  same  sequence  can  be  shown  by  holding  a  visual  trait  constant  for 
each  element. 


To  show  a  sequence,  the  “elements”  relation  is  shown  by  holding  constant  one  trait;  the  “or¬ 
dering”  relation  is  shown  by  varying  another  trait.  For  example,  the  vertical  position  can  be  kept 
constant  while  the  horizontal  varies,  shown  first  below,  or  the  role  of  the  horizontal  and  vertical 
reversed,  yielding  the  second  display  below. 


For  the  sequence  [1,2,3]: 


1  2  3 


1 

2 

3 


Two  other  association  possibilities  for  maps  are  shown  in  the  extended  example:  (a)  using  line 
segments  to  link  related  items,  and  (b)  using  a  plot  chart. 


Visual  Representations 


In  this  section  we  consider  in  more  detail  available  visual  representations  and  how  they  can  be 
used. 

To  display  a  set  of  relationships  adequately,  we  must  represent  every  such  relationship  visually 
and  not  have  any  other  relationship  implied  by  the  display. 

The  visual  representations  considered  in  this  paper  are  the  following: 


1.  A  horizontal  or  vertical  positional  association  can  represent  a  tuple,  a  sequence  or  a  set. 
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2.  A  connecting  line  can  represent  a  pair. 

3.  A  plot  chart  can  represent  a  map  A  X  B  *-*■  C. 

4.  A  box  may  represent  any  complex  object. 

5.  A  visual  attribute  (e.g.,  color,  shape,  or  font)  may  represent  a  pair. 


Note  that  we  consider  a  pair  to  be  a  special  case  of  a  tuple  (a  2-tuple). 

It  is  necessary  to  represent  the  objects  themselves  as  well  as  the  relationships  between  objects. 
In  this  paper  we  always  represent  objects  by  text  strings  (possibly  boxed). 

A  good  display  should  minimize  the  number  of  times  any  individual  object  is  represented.  How¬ 
ever,  it  is  often  desirable  to  represent  a  relationship  redundantly.  For  example,  in  diagrams  where 
relationships  are  always  represented  bv  lines,  it  is  desirable  to  represent  relationships  positionally  as 
well;  and,  in  complex  tables  where  sets  and  tuples  are  represented  positionally,  representing  them 
additionally  by  boxing  can  prevent  an  accidental  positional  alignment  from  being  misinterpreted 
as  an  additional  relationship. 


The  Relational  Language 

The  language  to  express  the  relational  information1  has  the  following  constructs: 

•  {x,  y, . . .}:  the  set  consisting  of  x,  y, . . . 

•  (x,  y, . . .):  the  tuple  consisting  of  x,  y, . . . 

•  [x,  y,  - .  .J :  the  collection  consisting  of  x,  y, . . . 

•  {f(x)\p(x)}:  the  set  consisting  of  /(x)  for  every  x  such  that  p(x), 

•  {l/(z)  *-*  g(x)|p(x)|}:  the  (partial)  map  that  maps  /(x)  to  g(x)  for  every  x  such  that  p{x). 

•  "a-string":  the  string  a-string. 

A  collection  is  different  from  a  set  in  that  all  the  elements  need  to  be  displayed  but  they  do  not 
have  to  be  visually  associated.  For  example,  the  elements  of  a  set  must  be  displayed  consistently, 
such  as  in  a  single  row  or  column,  whereas  the  elements  of  a  collection  may  appear  at  any  relative 
position  on  a  page  or  on  different  pages. 

The  following  is  an  example  specification  of  relational  information: 


'The  language  does  not  include  relations  as  a  primitive  type;  instead,  relations  are  modeled  as  sets  of  tuples. 


[(  ’Symptom",  {x  j  (x,  y)  in  Diagnosis- Relation}), 

( "Fault  ,  { y  |  (x,  y)  in  Diagnosis- Relation}), 

(’Bus  Problems  ,  {(x,  y)  \  (x,  y)  in  Diagnosis- Relation})} 

Considered  as  a  specification  of  information  to  be  visually  displayed,  this  structure  can  be  read 
as  follows.  Display  three  structures:  first,  a  tuple  with  the  first  element  (the  label)  Symptom,  and 
with  the  second  element  being  the  set  of  all  x,  whenever  ( x,y )  is  in  the  set  Diagnosis-Relation; 
second,  a  tuple  with  first  element  Fault ,  and  with  second  element  being  the  set  of  all  y ,  whenever 
(x,  y)  is  in  the  set  Diagnosis- Relation;  and  third,  a  tuple  with  first  element  Bus  Problems ,  and  with 
second  element  being  the  set  of  all  tuples  (x,y),  whenever  (x,y)  is  in  the  set  Diagnosis-Relation. 

The  set  comprehension  in  the  third  tuple  is  just  the  set  Diagnosis- Relation.  This  expanded 
form  is  used  to  explicate  the  structure  of  the  relation  so  it  may  be  transformed  by  simple  structural 
transformation  rules. 

The  value  of  the  Diagnosis-Relation  that  we  will  use  in  our  examples  is: 

Diagnosis-Relation  = 

{(  Bad  data  received  ,  ’defective  board  ), 

(  Bad  data  received  ,  bad  sensor  ), 

("Transmission  fails  ,  "card  in  wrong  slot”), 

("Transmission  fails  ,  "defective  board  ), 

("Bus  error  signal  ,  ’card  in  wrong  slot  )} 

Note  that  this  specification  of  Diagnosis- Relation  is  concrete  in  that  it  does  not  have  any 
variables  in  it,  whereas  the  prior  example  is  abstract  in  that  it  depends  on  the  variable  Diagnosis- 
Relation.  We  will  use  these  specifications  in  our  examples  in  the  next  section. 


Tables  and  Diagrams 

The  system  we  have  implemented  takes  specifications  of  relational  information  in  the  language 
presented  in  the  previous  section  and  displays  them  so  as  to  reveal  their  structure.  Our  system 
has  two  subsystems:  (a)  one  for  generating  tables,  and  (b)  one  for  generating  diagrams.  The 
table  generator  works  with  abstract  specifications,  and  thus  produces  display  structures  are  for  the 
most  part  independent  of  the  actual  values  of  the  variables  (i.e.  they  are  data-independent).  The 
diagram  generator  works  with  concrete  specifications  and  thus  produces  data-dependent  displays. 
In  particular,  the  diagram  generator  works  with  binary  relations  where  all  the  structure  is  in  the 
data. 

The  table  generator  has  an  interpreter  that  takes  an  abstract  specification  and  produces  a 
display  with  the  same  structure  as  the  evaluated  specification.  Thus,  any  repetition  of  objects 
in  the  evaluated  specification  will  cause  repetition  in  the  display.  If  the  abstract  specification  is 
transformed  to  remove  a  redundancy,  then  the  display  derived  from  the  transformed  specification 
will  have  that  redundancy  removed. 

The  diagram  generator  displays  relations  using  links  to  connect  the  related  objects.  Each 
object  and  link  is  only  displayed  once  so  the  generator  is  not  concerned  with  removing  redundancy. 
Instead  it  is  concerned  with  layout  to  allow  simple,  short  links  as  much  as  possible,  and  to  allow 
redundant  use  of  positional  adjacency  to  strengthen  the  visual  association  for  the  viewer.  Indeed. 
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if  two  associated  objects  are  positionally  adjacent,  then  a  simple  link  may  be  drawn  between  them, 
so  these  two  criteria  are  complementary. 


Thus,  the  table  generator  is  primarily  working  on  redundancy  removal  and  has  simple  layout 
strategies,  whereas  the  diagram  generator  is  primarily  working  on  layout  and  need  not  be  concerned 
with  redundancy  removal. 

III.  TABLE  SYNTHESIS 


The  table  synthesizer  uses  three  kinds  of  rules:  (a)  representation  rules,  (b)  redundancy  removal 
rules,  and  fc)  conditioning  rules  applied  specifically  to  enable  application  of  rules  of  the  first  two 
kinds. 


Representation  Rules 

In  addition  to  the  three  primary  kinds  of  visual  association  described  in  the  following  three 
subsections,  boxing  is  used  around  sets  and  tuples  except  at  the  top  level.  Boxes  containing  boxes 
are  given  thicker  lines  than  the  boxes  within  them.  This  representation  redundancy  is  sufficient  to 
nullify  accidental  positional  alignments. 

Positional  Association 


A  tuple  or  a  set  may  be  represented  by  horizontal  or  vertical  positional  association.  If  the 
parent  of  a  tuple  or  set  is  represented  by  horizontal  or  vertical  association  then  the  opposite  is 
chosen.  This  simple  rule  is  too  restrictive  in  general  but  is  sufficient  for  the  kinds  of  examples 
considered  in  this  paper.  The  top-level  choice  is  unconstrained  so  the  generator  will  generate  two 
identical  displays  with  x  and  y  positions  reversed. 

For  example,  if  the  top-level  of  the  specification 
{(x,  y)  |  (x,  y)  in  Diagnosis- Relation] 

is  represented  by  vertical  association  then  the  tuple  will  be  represented  by  horizontal  association, 
giving  the  first  display  in  Figure  2.  The  second  display  results  from  the  reverse  choice. 

We  will  show  henceforth  only  one  of  these  symmetrical  pairs  of  displays  in  each  case.  However, 
this  variety  is  important.  Sometimes  one  orientation  will  not  fit  on  a  page  but  its  symmetrical 
version  will.  Sometimes  it  is  more  convenient  for  a  user  to  locate  information  by  searching  down  a 
column  rather  than  across  a  row. 

Connecting  Line  Association 

A  set  of  pairs  may  be  represented  visually  by  displaying  the  set  of  first  elements  of  the  represen¬ 
tation,  the  set  of  second  elements  of  the  pairs,  and  a  line  for  each  pair  connecting  the  first  element 
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Bus  error  signal 

card  in  wrong  slot 

Transmission  fails 

defective  board 

Transmission  fails 

card  in  wrong  slot 

Bad  data  received 

bad  sensor 

Bad  data  received 

defective  board 

Bus  error  signal 

Transmission  fails 

Transmission  fails 

Bad  data  received 

Bad  data  received 

card  in  wrong  slot 

defective  board 

card  in  wrong  slot 

bad  sensor 

defective  board 

Figure  2.  Symmetrical  Displays  of  Relation 


of  the  pair  with  the  second  element.  This  representation  is  the  basis  of  the  diagram  generator.  We 
include  a  special  case  in  the  table  generator  that  does  not  require  data-specific  layout  (although 
it  may  benefit  from  it):  display  the  first  elements  vertically  and  next  to  them  display  the  second 
elements  vertically  with  space  in  between  them  where  the  connecting  lines  are  drawn.  It  is  useful 
if  the  set  of  first  elements  is  disjoint  from  the  set  of  second  elements  and  there  are  not  too  many 
line  crossings. 

The  rule  performs  the  transformation 
{(x,  y)  |  p(x,  y)} 

L{x  |  p(x,  y)},  {y  |  p(x,  y)},  {(x,  y)  |  p(x,  y)}J 

along  with  annotations  to  specify  the  display  methods  of  the  components. 

For  example,  the  specification 
{(x,  y)  |  (x,  y)  in  Diagnosis- Relation} 

is  transformed  to 

|_{x  |  (x,  y)  in  Diagnosis- Relation] , 

{y  |  (x,  y)  in  Diagnosis- Relation}, 

{(x,  y)  |  (x,  y)  in  Diagnosis- Relation}} 

resulting  in  the  display  of  Figure  3.  This  figure  illustrates  an  exception  to  the  simple  boxing  strategy 
described  above.  The  set  of  first  elements  and  the  set  of  second  elements  are  not  boxed  because  of 
the  lines  pointing  to  elements  of  the  sets. 
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Bad  data  received 
Transmission  fails 
Bus  error  signal 


bad  sensor 
defective  board 
card  in  wrong  slot 


Figure  3.  Line  Display 


Plot  Chart 


A  plot  chart  visually  represents  a  map  m  :  Ax  B  i->  C .  The  set  A  is  represented  vertically  and 
the  set  B  horizontally  (or  vice  versa),  and  placed  offset  so  the  values  of  m(a,  b)  for  every  a  £  A  and 
b  £  B  can  be  displayed  at  the  x  position  of  b  and  the  y  position  of  a.  See  Figure  4  for  an  example. 

The  transformation  has  a  similar  form  to  that  of  the  connecting  line  transformation  but  the 
annotations  specifying  layout  are  different: 

{(*,  y)  true  |  p(x,  y)} 

[_{*  I  y )},  { y  I  p(x,  y)},  {(*,  y)  true  I  p(x ,  y)}J 

For  example,  the  specification 
(I  (i,  y)  true  |  (x,  y)  in  Diagnosis- Relation  |} 

is  transformed  to 

[{x  |  (x,  y)  in  Diagnosis- Relation}, 

{y  I  (z>  y)  in  Diagnosis- Relation} , 

{|  (x,  y)  *-*  true  \  (x,  y)  in  Diagnosis- Relation  |}J 

resulting  in  the  display  of  Figure  4  (with  “X”  being  used  to  represent  the  value  "true”). 


bad  sensor 

defective  board 

card  in  wrong  slot 

Bad  data  received 

X 

X 

Transmission  fails 

X 

X 

Bus  error  signal 

X 

Figure  4. 

Plot  Chart  Display 
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Redundancy  Removal  Rules 


There  are  two  kinds  of  redundancy  removal  rules:  (a)  one  removes  common  subexpressions  by 
merging,  and  (b)  the  second  projects  on  a  field  of  a  relation  to  avoid  repetition  of  that  element  of  the 
relation.  We  give  an  example  of  (b)  following;  examples  of  (a)  will  be  presented  in  the  derivations  of 
more  complex  tables  below,  where  the  motivation  is  clearer.  Note  that  the  programming  language 
option  of  introducing  a  variable  to  hold  the  value  of  a  common  subexpression  is  not  as  simple  in  a 
visual  representation  and  is  not  used  in  our  system.  Instead  we  perform  a  merging  of  the  contexts 
in  which  the  common  subexpressions  appear. 

The  rule  that  projects  on  the  first  element  of  a  binary  relation  can  be  expressed  as 
{(*,  y)  I  P(x i  V )} 

{<*,  {y  I  Pix ,  y)})  I  p(x,  yl )} 

For  example,  the  specification 
{|  ( x ,  y)  |  (x,  y)  in  Diagnosis- Relation  |} 

is  transformed  to 

{(x,  {y  |  (x,  y)  in  Diagnosis- Relation}) 

|  (x,  yl)  in  Diagnosis- Relation] 

resulting  in  the  display  of  Figure  5. 


Bad  data  received 

defective  board 

bad  sensor 

Transmission  fails 

card  in  wrong  slot 

defective  board 

Bus  error  signal 

card  in  wrong  slot 

Figure  5.  Projection  Display 


Conditioning  Rules 

The  conditioning  rules  for  enabling  applicability  of  redundancy  removal  rules  are  described  in 
context  in  the  derivations  of  more  complex  tables  below.  The  only  conditioning  rule  for  enabling 
applicability  of  a  visual  representation  rule  that  we  require  in  this  paper  is  the  rule  that  converts 
a  set  to  a  boolean  map  (its  characteristic  function): 

{/(*)  I  p(x)} 

{|  f(x)  —  true  |  p( x )  |} 
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For  example,  the  specification 
{|  ( x ,  y)  |  { x ,  y)  in  Diagnosis- Relation  |} 

is  transformed  to 

{|  { x ,  y)  *— >•  true  |  (x,  j/)  in  Diagnosis- Relation  |} 
which  enables  the  plot  chart  representation  transformation. 


Generator  Structure 


The  display  synthesizer  creates  a  number  of  possible  displays  for  the  input  specification.  The 
generator  performs  a  number  of  data  reformulations  and  display  style  choices.  For  each  successful 
combination  of  choices,  a  display  program  is  synthesized.  Compiling  and  executing  the  display 
program  creates  the  screen  or  window  output. 

The  generator  actually  consists  of  two  generators  acting  in  series.  The  first  manipulates  the 
specification  into  a  variety  of  “equivalent”  forms.  For  each  form,  the  second  generator  cycles  through 
possible  choices  of  visual  display  styles.  Both  generators  use  top-down,  successive  refinement  on 
the  specification  and  its  subparts.  Each  generator  annotates  the  specification  with  various  display 
directives.  At  the  conclusion  of  the  design  generation  the  specification  is  fully  annotated.  Then 
the  code  synthesizer  recursively  descends  through  the  specification,  synthesizing  display  code  in 
accordance  with  the  directives. 

The  entirety  of  all  the  combinations  of  possible  choices  is  called  the  search  space.  The  specifics 
of  the  control  strategies  for  the  generators  determine  in  what  order  the  space  is  explored.  One 
motive  for  extending  our  theory  to  include  a  selection  component  is  to  reduce  the  number  of 
display  possibilities  that  must  be  fully  explored. 

Since  the  control  structure  is  independent  of  the  actual  transformations,  more  data  reformula¬ 
tions  and  display  styles  can  be  incrementally  added. 

In  the  next  three  sections  we  look  at  derivations  of  more  complex  tables. 


Bus  Problem  Diagnosis  Table 


In  this  example  we  consider  the  Diagnosis- Relation  given  above  but  with  labels  for  the  relation 
and  its  domain  and  range.  The  abstract  specification  for  this  is 

[("Symptom  ,  {x  |  (x,  y)  in  Diagnosis- Relation}) , 

(  Fault  ,  {y  |  (x,  y)  in  Diagnosis- Relation}), 

(  Bus  Problems  ,  {(x,  y)  |  (x,  y)  in  Diagnosis- Relation}) j. 

This  display  corresponding  to  this  specification  is  shown  in  Figure  6. 

Within  each  top-level  tuple,  the  first  and  second  elements  are  associated  horizontally.  The 
second  element  of  each  tuple  is  a  set,  the  elements  of  which  are  associated  vertically.  The  set 
elements  for  the  third  top-level  tuple,  which  are  themselves  tuples,  have  their  fields  associated 
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Symptom 


Bad  data  received 
Transmission  fails 
Bus  error  signal 


Fault 


bad  sensor 
defective  board 
card  in  wrong  slot 


Bus  Problems 

Bus  error  signal 

card  in  wrong  slot 

Transmission  fails 

defective  board 

Transmission  fails 

card  in  wrong  slot 

Bad  data  received 

bad  sensor 

Bad  data  received 

defective  board 

Figure  6.  Diagnosis:  Initial  Display 


horizontally.  The  top-level  tuples  are  presented  vertically  one  after  the  other  as  a  default,  although 
the  specification  does  not  demand  this. 


Expression  Embedding 

In  the  display  in  Figure  6  the  domain  and  range  are  completely  redundant  because  their 
elements  all  occur  in  the  relation.  To  remove  this  redundancy  the  domain  and  range  expressions 
cannot  simply  be  deleted  because  the  association  with  the  domain  and  range  labels  would  be  lost. 
These  associations  can  be  maintained  by  replacing  the  elements  in  the  relation  by  label-element 
pairs,  after  which  the  domain  and  range  expressions  can  be  safely  omitted  along  with  their  label 
associations.  The  resulting  specification  is 

("Bus  Problems", 

{(("Symptom",  x),  ("Fault",  y)) 

|  (x,  y )  in  Diagnosis- Relation}) 

and  the  resulting  display  is  shown  in  Figure  7. 

This  transformation  has  traded  off  redundancy  in  domain  and  range  elements  with  the  redun¬ 
dancy  of  repeated  label  appearances.  In  some  cases,  this  repetition  might  be  useful.  For  example, 
if  a  program  variable  was  present  in  multiple  packages,  it  would  be  important  to  print  the  package 
name  to  be  clear  about  which  variable  is  being  referenced.  Or  with  some  applications  the  repetition 
might  be  mandated — if  the  heading  was  a  formal  title  such  as  the  “The  Honorable”.  Another  option 
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Bus  Problems 


Symptom 

Fault 

Bus  error  signal 

card  in  wrong  slot 

Symptom 

Fault 

Transmission  fails 

defective  board 

Symptom 

Fault 

Transmission  fails 

card  in  wrong  slot 

Symptom 

Fault 

Bad  data  received 

bad  sensor 

Symptom 

Fault 

Bad  data  received 

defective  board 

Figure  7.  Diagnosis:  After  Embedding  Domain  and  Range 

is  to  encode  the  pairing  using  color,  shape,  or  font,  so  less  display  space  is  used.  This  option  is  not 
yet  implemented  in  our  system.  The  next  section  shows  how  the  redundancy  can  be  removed. 


Constant  Extraction 


We  wish  to  avoid  the  redundancy  of  the  repetition  of  “Symptom”  and  “Fault"  introduced  by 
the  previous  transformation.  This  repetition  is  reflected  in  the  current  specification  as  the  constant 
strings  being  within  a  repeated  part  of  the  set  comprehension: 

("Bus  Problems", 

{(("Symptom",  x),  ("Fault",  y)) 

|  (x,  y)  in  Diagnosis- Relation}) . 

To  avoid  this  repetition  we  must  isolate  the  constants  “Symptom"  and  “Fault"  from  the  vari¬ 
ables  x  and  y.  The  conditioning  rule  for  this  purpose  applies  to  the  tuple 

(("Symptom",  x),  ("Fault",  y)) 
producing 

(("Symptom",  "Fault"),  (x,  y)). 

This  transformation  is  a  transposition  if  we  interpret  the  tuple  of  tuples  as  a  matrix.  However, 
visual  representation  does  not  guarantee  this  interpretation  and  other  interpretations  may  not 
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preserve  the  association  of  “Symptom”  with  z  and  “Fault”  with  y.  Therefore,  the  rule  adds  an 
annotation  to  constrain  the  visual  interpreter  to  make  this  association. 

After  this  application  the  specification  is 

(  ’Bus  Problems  ", 

{((''Symptom”,  "Fault"),  (z,  y)) 

|  (z,  y)  in  Diagnosis- Relation}}. 

With  this  form  we  may  project  the  relation  on  its  first  element  and  thus  avoid  repetition  of  the 
labels: 

("Bus  Problems  ", 

{((  Symptom  ,  Fault  ),  {(z,  y)  |  (z,  y)  in  Diagnosis- Relation}) 

|  ( xl ,  yl)  in  Diagnosis- Relation}). 

When  Diagnosis-Relation  is  non-empty,  this  simplifies  to 

("Bus  Problems', 

{(("Symptom",  "Fault"', 

{(z,  y)  |  (z,  y)  in  Diagnosis- Relation}))} . 

Displaying  a  singleton  set  is  considered  to  be  the  same  as  displaying  just  the  single  element  of 
the  set.  Therefore,  this  is  the  same  as 

(  "Bus  Problems", 

(( "Symptom",  "Fault"), 

{(z,  y)  |  (z,  y)  in  Diagnosis- Relation})). 

The  resulting  display  is  shown  in  Figure  8. 


Bus  Problems 


Symptom 

Fault 

Bus  error  signal 

card  in  wrong  slot 

Transmission  fails 

defective  board 

Transmission  fails 

card  in  wrong  slot 

Bad  data  received 

bad  sensor 

Bad  data  received 

defective  board 

Figure  8.  Diagnosis:  After  Removing  Label  Repetition 
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Avoiding  Repetition  Of  Domain  Elements 


The  remaining  redundancies  in  this  display  are  repetitions  of  the  symptom  and  fault  names. 
The  repetition  of  the  symptoms  is  remedied  by  projecting  on  the  first  element  of  the  tuple 

("Bus  Problems”, 

(("Symptom”,  "Fault”), 

{(x,  {y  |  ( x ,  y)  in  Diagnosis- Relation}) 

|  (x,  yl)  in  Diagnosis-Relation })) 

giving  the  display  in  Figure  9. 


Bus  Problems 


Symptom 

Fault 

Bad  data  received 

defective  board 

bad  sensor 

Transmission  fails 

card  in  wrong  slot 

defective  board  | 

Bus  error  signal 

card  in  wrong  slot  1 

Figure  9.  Diagnosis:  After  Domain  Projection 

This  display  still  has  a  repetition  of  fault  names  but  there  is  no  simple  transformation  that  can 
remove  this  repetition.  Instead  of  projecting  on  the  first  element  of  the  tuple  to  avoid  repetition  of 
symptoms,  we  could  have  projected  on  the  second  elements  to  avoid  repetition  of  faults  but  then 
we  would  have  had  repetition  of  symptoms.  We  need  to  return  to  the  original  specification  and  use 
different  transformations  to  avoid  repetition  of  either. 


Connecting  Line  Table 


The  original  specification  is 

[("Symptom  .  {x  [  (x,  y)  in  Diagnosis- Relation}) , 

(  Fault  ,  {y  |  (x.  y)  in  Diagnosis- Relatioji}), 

{  Bus  Problems  ,  {(x,  y)  |  (x,  y)  in  Diagnosis- Relation))}. 

The  connecting  line  representation  transformation  applies  to  the  final  set  comprehension  giving 

[("Symptom  .  (x  |  (x,  y)  in  Diagnosis- Relation}), 

(  Fault  ,  {y  |  (x.  y)  in  Diagnosis- Relation}) , 

("Bus  Problems”, 

[{x  |  (x,  y)  in  Diagnosis- Relation}, 
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{y  |  ( x ,  y)  in  Diagnosis- Relation}, 

{(x,  y)  |  ( x ,  y)  in  Diagnosis- Relation) \)\. 

Now  a  common  subexpression  merging  rule  applies.  The  rule  detects  the  set  of  first  elements 
from  the  Diagnosis- Relation  is  the  same  set  it  was  asked  to  display  in  association  with  “Symptom” . 
And  the  set  of  second  elements  has  the  same  value  as  the  set  associated  with  “Fault”.  After 
identifying  these  common  terms,  the  specification  is  rewritten  to  use  each  only  once: 

("Bus  Problems", 

L(  Symptom  ,  {x  |  ( x ,  y)  in  Diagnosis- Relation}), 

("Fault  ,  {y  |  (x,  y)  in  Diagnosis- Relation}) , 

{(x,  y)  |  (x,  y)  in  Diagnosis- Relation}}). 

The  resulting  display  is  shown  in  Figure  10. 


Bus  Problems 


Symptom 

Fault 

Bad  data 

bad  sensor 

received  \ 

Transmission 

defective 

fails  \ 

board 

Bus  error 

card  in 

signal 

wrong  slot 

Figure  10. 

Diagnosis:  Line  Table 

Plot  Chart 


Before  the  plot  chart  representation  rule  can  apply,  we  must  return  to  the  original  specification 
and  then  apply  to  the  finai  set  comprehension  the  conditioning  rule  that  converts  a  set  to  a  boolean 
map.  giving 

[("Symptom  ,  {x  |  (x,  y)  in  Diagnosis- Relation}) . 

(  Fault  ,  {y  |  (x,  y)  in  D lugnosis- Relation}) , 

(  Bus  Problems  ,  {|  (x,  y)  >-*  true  |  (x,  y)  in  Diagnosis- Relation  |})J. 

The  plot  chart  rule  now  applies  to  this  map  expression  giving 

[(  Symptom  ,  {x  |  (x,  y)  in  Diagnosis- Relation}) , 

(  Fault  ,  {y  |  (x,  y)  in  Diagnosis- Relation}), 

(  Bus  Problems  , 

[{x  |  (x,  y)  in  Diagnosis- Relation} , 

{y  I  (*,  y )  Diagnosis- Relation} , 

{|  (x,  y)  <—  true  |  (x,  y)  in  Diagnosis- Relation  |}J)J. 
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with  appropriate  annotations. 


The  same  redundancy  removal  rule  that  merged  the  common  subexpressions  in  the  line  table 
derivation  also  applies  here,  giving 

("Bus  Problems  ’, 

[{’’Symptom  ,  {x  |  (x,  y)  in  Diagnosis-  Re  la  tion}) , 

{”Fault  ,  {y  |  { x ,  y)  in  Diagnosis- Relation}), 

{|  ( x ,  y)  |  (x,  y)  in  Diagnosis- Relation  |}J). 

The  resulting  display  is  shown  in  Figure  11. 


Bus  Problems 


Fault 

bad  sensor  defective  card  in 

board  wrong  slot 

Symptom 

Bad  data 

received 

Transmission 

fails 

Bus  error 
signal 

X  X 

X  X 

X 

Figure  11.  Diagnosis:  Plot  Chart. 

The  entries  in  the  map  itself  are  given  the  sameness  plot-chart.  Referring  to  the  map  entries, 
(x,  y)  h-  z  the  meaning  of  plot-chart  is  that  the  z  range  values  are  to  be  positioned  in  the  same 
column  as  the  x  value  appears,  and  the  same  row  as  the  y  value.  In  order  to  do  that,  the  system 
must  keep  track  of  the  row  and  column  coordinates  as  it  prints  the  domain  and  range  sets.  The 
Domain-Colm-Map  and  Range-Row-Map  are  created  by  the  transformation  to  hold  this  information. 

After  this  transformation  finishes  the  specification  is  annotated  with  the  visual  display  choices, 
and  the  auxiliary  data  structures  have  been  created.  Then  the  synthesis  of  the  actual  display  code 
begins.  First,  the  code  for  printing  the  domain  and  range  sets  is  synthesized.  Within  this  code,  the 
x  and  y  coordinate  values  for  the  Domain-Colm-Map  and  Range- Row-Map  will  be  stored.  Then 
the  code  for  the  table  entries  is  synthesized  by  the  function  Plot-Chart. 

The  important  action  taken  by  this  synthesis  step  is  to  determine  the  coordinates  for  printing 
the  value  of  z  by  setting  the  current-colm  to  be  the  column  for  x  from  the  Domain-Colm-Map.  and 
the  current-row  to  be  the  row  containing  y  from  the  Range- Row- Map. 
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Example:  Muiticolumn  Table 

This  example  shows  the  generation  of  a  table  with  more  than  two  columns.  The  data  is  a  4-ary 
relation: 

Best-Options  = 

{{  "Replace  Low  Power  RF  Unit  ",  1.9,  ”30%  ,  "9473A4"), 

("Repair  LPRF/Computer  Cable  ,  0.4,  "20%  ,  "N/A  ), 

("Replace  FCR  Computer",  1.2,  ".50%  ,  "9473A6"), 

("Check  LPRF/Computer  Cable",  0.4,  "20%",  "N/A")}. 

The  initial  description  of  the  data  to  be  displayed  is 

[("Description",  {  W  \  ( IV,  x,  y,  Z)  in  Best-Options}), 

("Hours  ,  {r  |  (IF,  x,  y,  Z)  in  Best- Options}), 

("Fail  Prob",  {y  |  (IF,  x,  y,  Z)  in  Best- Options}), 

("Ref  Des",  {Z  |  ( W,  x,  y,  Z)  in  Best-Options}) , 

("Best  Options",  {(IF,  x,  y,  Z)  |  (IF,  x,  y,  Z)  in  Best-Options})} . 

The  same  initial  transformations  as  for  the  diagnosis  example  apply  giving 
("Best  Options", 

(("Description",  "Hours",  "Fail  Prob",  "Ref  Des"), 

{(IF,  x,  y,  Z)\{W,  x,  y,  Z)  in  Best- Options})) 

which  produces  the  display  in  Figure  12. 


Best  Options 


Description 

Hours 

Fail  Prob 

Ref  Des 

Check  LPRF/Computer  Cable 

0.4 

20% 

N/A 

Replace  FCR  Computer 

1.2 

50% 

9473A6 

Repair  LPRF/Computer  Cable 

0.4 

20% 

N/A 

Replace  Low  Power  RF  Unit 

1.9 

30% 

9473A4 

Figure  12.  Multi 

-Column 

Table 

Because  the  relation  is  not  binary,  fewer  of  the  other  transformations  apply  and  none  lead  to 
any  other  interesting  displays. 


Example:  Complex  Plot  Chart 

This  example  shows  how  plot  charts  sharing  a  common  domain  may  be  combined.  The  original 
specification  is 
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[{"Faults",  {F  j  (F,  S,  T5,  R)  in  Fault-Info}), 

("Symptoms",  {5  |  {F,  S,  TS,  R)  in  Fault-Info}), 

("Tests",  {  TS  |  {F,  S,  TS,  R)  in  Fault-Info}), 

("Repairs",  {R  j  (F,  S,  TS,  ft)  in  Fault-Info}), 

("Fault  Information  , 

{(F,  S)  ]  { F ,  S,  TS,  R)  in  Fault-Info}, 

{(F,  TS)  |  (F,  S,  TS,  R)  in  Fault-Info}, 

{(F,  R)  |  (F,  S,  TS,  R)  in  Fault-Info}) J. 

There  are  many  displays  the  table  generator  produces  for  this  specification.  The  only  ones 
introducing  new  ideas  are  the  ones  combining  plot  charts  for  the  relations,  so  only  the  applications 
of  the  common  subexpression  merging  rule  leading  to  the  combined  charts  are  shown. 

After  two  of  the  relations  have  been  transformed  into  maps  and  represented  as  plot  charts,  and 
one  of  them  has  had  its  domain  and  range  embedded  in  it,  we  have 

[{"Symptoms",  {5  |  (F,  S,  TS,  R)  in  Fault- Info}) , 

("Tests",  {TS  |  (F,  S,  TS,  R)  in  Fault-Info}), 

("Fault  Information  , 

{{F,  S)  |  (F,  S,  TS,  R)  in  Fault-Info}, 

[{F  j  {F,  S,  TS,  R)  in  Fault-Info}, 

{  TS  |  (F,  S,  TS,  R)  in  Fault-Info}, 

{|  {F,  TS)  *-*  true  |  (F,  S,  TS,  ft)  in  Fault-Info  |}J, 

[("Faults",  {F  |  {F,  S ,  TS,  R)  in  Fault-Info }), 

("Repairs",  {ft  |  (F ,  S,  TS,  ft)  in  Fault-Info}), 

{|  (F,  ft)  i~*  true  |  (F,  S,  TS,  ft)  in  Fault-Info  [}J)j. 

The  range  is  embedded  as  in  previous  derivations  but  the  repeated  domains  require  a  more 
subtle  merging  so  as  not  to  lose  information.  The  result  is 

[("Symptoms  ,  {5  |  {F,  S ,  TS,  ft)  in  Fault-Info }), 

(  Fault  Information  , 

{{F,  S)  |  {F,  S,  TS,  ft)  in  Fault-Info}, 

[("Faults",  {F  |  (F,  5,  TS,  ft)  in  Fault-Info}), 

(("Tests",  {  TS  |  (F,  S,  TS,  ft)  in  Fault-Info}), 

("Repairs",  {ft  |  (F,  S,  TS,  ft)  in  Fault- Info})), 

[{|  (F,  TS)  true  |  (F,  5,  TS,  ft)  in  Fault-Info  |}, 

{|  (F,  ft)  h-*.  true  |  (F,  S,  TS,  ft)  in  Fault-Info  |}JJ)J. 

After  transforming  the  third  relation  into  a  map  and  representing  it  as  a  plot  chart,  the  same 
common  subexpression  merging  process  produces 

("Fault  Information  , 

(("Faults", 

{F  |  (F,  S,  TS,  ft)  in  Fault-Info}), 

(("Symptoms",  {5  |  (F,  S,  TS,  ft)  in  Fault- Info}) , 

({'"Tests",  j  TS  |  (F,  5,  TS,  ft)  in  Fault-Info}), 

("Repairs'  ,  {ft  |  (F,  S,  TS,  ft)  in  Fault- Info}))), 

[{j  (F,  S )  true  |  (F,  S,  TS,  ft)  in  Fault-Info  j}, 

{|  (F,  TS)  >-*■  true  |  (F,  S,  TS,  ft)  in  Fault-Info  |}, 

{|  (F,  ft)  >-♦  true  j  (F,  S,  TS,  ft)  in  Fault-Info  |}J)). 
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which  gives  the  display  in  Figure  13.  The  simple  boxing  strategy  used  does  not  work  very  well  here 
because  it  does  not  recognize  an  incidental  nesting  of  range  labels  and  sets.  There  is  a  box  missing 
from  the  entire  table  because  only  three  levels  of  boxes  are  allowed. 
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Figure  13.  Combined  Plot  Chart 


IV.  DIAGRAM  SYNTHESIS 


This  section  describes  the  prototype  diagram  synthesis  system  we  have  developed.  The  overall 
goal  we  are  pursuing  is  a  coherent  and  uniform  presentation  of  functional  data.  In  contrast  to 
table  generation,  the  principle  problem  is  constructing  a  pleasing  and  appropriate  layout  of  all  the 
diagram  components  and  consonant  drawing  of  any  connecting  arcs. 

The  first  subsection  below  presents  a  brief  overview  of  the  issues  involved  in  diagram  synthesis 
and  describes  our  general  approach.  The  next  subsection  discusses  design  foundations  of  the  present 
system;  the  following  one  examines  in  more  detail  the  layout  strategies  currently  implemented. 
Several  example  diagrams  synthesized  by  the  system  are  included  in  Section  IV  as  well. 


Overview 


As  with  table  synthesis,  we  seek  to  represent  functional  similarities  and  interconnections  as 
visual  ones.  Since  the  diagrams  we  construct  are  presently  limited  to  boxes  and  lines  (the  lines 
represent  functional  or  dependency-based  connections),  this  goal  primarily  amounts  to  arranging 
and  rendering  the  diagram’s  components  so  that  such  similarities  are  immediately  conveyed. 


The  Issues 


In  general,  we  consider  the  problem  of  producing  a  pleasing  diagram  layout  as  one  of  constraint 
satisfaction ,  as  in  (Voigt  h  Tong,  1989)  or  (Braudaway,  1989).  A  number  of  issues  that  may  be 
viewed  as  constraints  enter  into  the  construction  of  a  graphical  representation  for  relational  data. 
Chief  among  these  are  overall  organization  and  space  management — the  resulting  diagram  must 
provide  an  easily  assimilated  presentation  of  the  intended  interrelationships,  and  will  normally  be 
required  to  fit  within  the  boundaries  of  some  predetermined  space.  These  basic  goals  can  often 
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conflict,  since  a  natural  way  to  emphasize  functional  distinction  and  also  reduce  visual  complexity 
is  to  space  elements  further  apart. 


Another  way  to  emphasize  similarity  or  distinction  is  through  the  use  of  colors,  shading,  and 
font  selection  for  component  labels.  We  do  not  presently  make  heavy  use  of  different  colors  and 
fonts,  though  we  do  support  several  varieties  of  shading  to  indicate  the  current  maintenance  status 
of  a  given  component.  These  may  be  selected  and  modified  directly  through  pop-up  menu  selections 
within  the  user  interface. 

For  the  IMIS  application,  we  need  to  synthesize  diagram  presentation  packages  that  can  be 
executed  through  different  graphics  utilities  on  a  variety  of  display  hardware.  Presentation  on  a 
hand-carried  portable,  for  instance,  will  impose  more  severe  size  constraints  than  on  a  large-screen 
work-station.  As  another  example,  presentation  on  paper  media  will  generally  not  (yet)  involve  the 
use  of  color  and  may  also  restrict  the  availability  of  different  fonts. 


General  Approach 

We  expect  to  receive  abstract  relational  input,  without  explicit  formatting  or  layout  informa¬ 
tion,  plus  high-level  forms  of  guidance  regarding  desired  properties  of  the  diagram  to  be  constructed . 
Such  guidance  can  simply  be  a  selection  and/or  ordering  of  particular  synthesis  strategies  to  apply. 
Layout  constraints  could  also  take  the  form  of  logic-based  rules  extracted  from  the  semantics  of  the 
entities  being  represented. 

Human  factor  constraints  provide  another  class  of  layout  guidance  (Ding  &  Mateti,  1990), 
(Gunning,  1987).  These  can  range  from  limits  on  the  minimum  text  fonts  or  on  the  number  of 
different  elements  presented,  to  color  selections  and  maximum  allowed  angles  within  a  link  (con¬ 
nection)  path.  Target  display  hardware  capabilities  or  limitations  must  also  be  taken  into  account. 
These  might  be  thought  of  as  machine-factor  constraints. 

We  construct  a  given  layout  through  incremental  assignment  of  various  positioning  attributes 
to  diagram  objects.  This  positioning  is  described  according  to  a  geometric  model  defined  within  the 
Refine™  knowledge  base.  A  number  of  coordinate  and  direction  transformations  are  supported. 

Positioning  of  all  diagram  objects  is  expressed  through  relative  coordinates  until  the  entire 
diagram  has  been  constructed.  Then  absolute  positions  can  be  assigned,  based  upon  the  size  of  the 
entire  diagram  and  its  placement  within  a  surrounding  screen  or  page  coordinate  system.  Finally, 
connecting  links  will  be  laid  out  to  complete  the  diagram. 

A  backtracking  control  structure  cycles  through  different  layout  possibilities,  pausing  to  display 
each  completed  diagram  which  passes  all  current  constraints.  The  user  can  stop  the  generation 
sequence  at  any  time  to  edit  the  current  diagram,  adding  or  deleting  components  and  their  inter¬ 
connections  with  the  mouse.  A  new  generation  sequence  can  then  be  invoked,  with  new  constraints 
and  selection  of  different  strategies  to  apply,  if  so  desired. 
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Design  Foundations 


Generation  and  Selection 


As  discussed  in  Section  II,  the  diagram  synthesis  process  may  be  viewed  as  comprising  a  gen¬ 
eration  and  a  selection  part.  The  generator  produces  a  series  of  box  arrangements  and  connecting 
line  placements  from  which  the  selector  chooses  a  subset  to  be  presented  as  complete  drawings.  At 
present,  the  selector  makes  its  choices  only  on  the  basis  of  some  simple  measures  of  connecting  link 
(line)  complexity  and  aesthetics  (Ding  &  Mateti,  1990). 

Similar  to  the  table  generation  control  structure  outlined  in  Section  III,  the  diagram  generation 
control  structure  also  consists  of  two  generators  acting  in  series.  In  diagram  synthesis,  the  first 
generator  combines  transformations  of  the  initial  input  relation  and  subsequent  construction  of  a 
layout  plan.  A  given  set  of  transformations  and  plan  construction  rules  corresponds  to  a  particular 
layout  strategy. 

For  each  strategy  applied,  the  second  generator  recursively  steps  through  the  layout  plan,  pro¬ 
ducing  a  number  of  potential  placements  for  each  diagram  component.  Whenever  the  sequence  of 
placement  choices  leads  to  a  plausible  total  arrangement,2  derivation  and  positioning  of  line  seg¬ 
ments  to  represent  component  linkage  completes  the  display  instance.  The  current  implementation 
constructs  the  single  “best”  set  of  link  lines  for  a  given  arrangement,  again  following  the  layout 
plan  and  backtracking  to  revise  earlier  choices  when  it  encounters  a  conflict  (such  as  a  line  crossing 
or  an  attempt  to  draw  through  a  component  box). 


Bounding  Box  Abstraction 


Both  the  diagram  and  table  synthesis  systems  use  a  hierarchy  of  bounding  boxes  to  allocate 
and  manage  screen  real  estate.  Each  box  represents  a  rectangular  region  surrounding  one  or  more 
components  of  a  diagram  (or  table). 

The  box  hierarchy  forms  a  tree,  in  which  a  box  surrounding  the  entire  diagram  corresponds 
to  the  root  and  individual  components  form  the  leaves.  Boxes  around  groups  of  components  cor¬ 
respond  to  inner  branches,  and  permit  diagram  subgroups  and  their  components  to  be  laid  out 
independently.  Such  groupings  not  only  enforce  visual  association  among  similar  objects  but  help 
combat  combinatorial  explosion  as  well.  The  dashed  outlines  in  Figure  14  show  the  bounding  boxes 
in  a  simple  diagram. 

Currently,  a  bounding  box  hierarchy  is  created  just  after  the  layout  plan  has  been  constructed. 
This  fixed  hierarchy  is  used  for  all  the  box  placements  generated  from  the  associated  layout  plan. 
In  future  versions,  we  envision  failures  to  satisfy  top-level  size  or  link-path  constraints  will  lead  to 
changes  in  the  placement  of  some  diagram  components  and  attendant  inner  restructuring  of  the 
box  hierarchy. 

2 For  now,  a  plausible  arrangement  is  just  one  in  which  no  boxes  overlap. 
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Figure  14.  Bounding  Box  Hierarchy  in  a  Simple  Diagram. 


Geometry  Model 

Individual  diagram  components  and  subgroups  are  positioned  relative  to  each  other  through  the 
use  of  a  two-dimensional  geometry  that  builds  upon  the  box  abstraction.  The  geometry  permits 
a  wide  range  of  possible  alignments  along  directed  offset- vectors  of  arbitrary  length.  Figure  15 
illustrates  the  available  alignment  points  and  offset-vectors  supported. 

Top  Top  Top 

Left  Center  Right 


Left 

Center 


Bottom  Bottom  Bottom 

Left  Center  Right 

Figure  15.  Offset  Directions  and  Box  Alignment  Points. 

In  the  most  general  case,  a  box  is  positioned  with  respect  to  another  box  through  specification 
of  an  offset-vector,  its  magnitude,  and  a  pair  of  alignment  points,  one  for  each  box. 

For  example,  positioning  a  box  some  positive  distance  d  directly  to  the  right  of  an  already 
placed  box  corresponds  to  specifying  a  Right  offset-vector  of  magnitude  d  and  the  alignment  pair 
(Right  Center ,  Left  Center).  That  is,  the  new  box  will  be  positioned  such  that  its  left-center  point 


will  lie  d  units  to  the  right  of  the  right-center  point  of  the  already  placed  box.  This  is  precisely 
the  alignment  illustrated  in  Figure  14,  in  which  the  bounding  box  containing  B  and  A  has  been 
positioned  with  respect  to  component  box  C. 

In  the  future,  we  expect  to  use  the  geometry  model  to  enforce  additional  placement  constraints 
specified  by  the  user,  such  as  “A  Left-Of  C,”  or  “A  Above  B.”  Such  positioning  relations  will  also 
be  derived  from  incorporation  of  flow  information,  classification  of  components  as  either  input  or 
output  elements,  and  so  forth. 


Initial  Layout  Strategies 

This  subsection  discusses  two  of  the  initial  layout  strategies  we  have  developed  and  imple¬ 
mented.  As  introduced  above,  a  layout  strategy  encompasses  (a)  transformations  on  the  original 
input  relation,  (b)  construction  and  use  of  a  layout  plan ,  and  (c)  constraints  and/or  preference 
orderings  on  the  particular  box  placements  generated  within  the  plan.  We  begin  with  a  brief  de¬ 
scription  of  the  input  transformations,  and  conclude  with  a  look  at  how  the  connecting  links  are 
constructed. 


Connection- based  Input  Transformations 

The  primary  input  to  the  diagram  system  is  simply  a  relation  specifying  the  links  or  connections 
existing  between  the  various  components.  More  precisely,  the  current  input  formalism  is  a  set  5  of 
sets  sj, . . . ,  s„,  where  each  s,  is  a  set  of  diagram  components  and 

{x,2/}  C  s ,  for  some  s,  €  S  =>  connected(x,y). 

For  example,  the  input  specification  for  the  diagram  in  Figure  14  was  just  {  {A,B,C}  }.  As  a 
notational  convenience,  the  text  strings  labels  for  each  component  may  be  specified  separately,  as 
a  map,  say  {|  A  — ►  "alpha",  B  — *  "beta",  C  —*  gamma"  |}. 

In  the  absence  of  additional  information,  individual  components  are  collected  into  subgroups 
solely  on  the  basis  of  their  connectedness.  We  apply  several  transformations  to  the  original  input 
relation,  producing  a  standard  form  that  isolates,  for  example,  the  most  or  least  connected  elements, 
and  collects  remaining  elements  with  similar  connectivity  into  such  subgroups.  These  subgroups 
will  eventually  be  placed  within  bounding  boxes  and  laid  out  in  a  uniform  way. 

The  transformed  relation  and  its  internal  subgroups  are  also  sorted  in  terms  of  the  total  number 
of  connections  for  each  element  and  subgroup.  Depending  on  the  current  layout  strategy,  these  sorts 
may  place  more  connected  components  before  less  connected  ones,  or  vice-versa. 


Orthogonal  “Center-Out”  Strategy 

The  basic  idea  in  the  Center-Oxit  strategy  is  to  first  fix  a  position  for  the  most  connected 
component(s)  and  then  distribute  the  remaining  diagram  elements  around  it.  This  notion  is  applied 
recursively  to  the  subgroups  of  the  diagram  as  well.  The  general  presentation  theory  at  work  here  is 
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that  placing  the  item  with  the  largest  number  of  emanating  links  first,  in  the  center,  will  generally 
lead  to  fewer  conflicts  in  eventual  link- path  construction. 

For  example,  the  input  relation 

{  {/cr,  cm},  {fee,  re,  co,  cw},  {rc,  ss,  tg,  cw}  }  , 

with  associated  text  labels  map 


rc  — *"  Radar  Control  Panel”, 

fee  FCC", 

tg  Throttle  Grip”, 

re  — REO", 

ss  Side  Stick  Controller  , 

co  — "  Cool”, 

cw  — <■”  Control  Wiring  , 

fer  FCR" 

, 

which  represents  components  of  a  radar  control  subsystem,  finds  a  number  of  pleasing  layouts  under 
this  strategy. 

After  application  of  the  input  transformations,  the  relation  has  the  form 

[  ( M,  [  [tg,  ss,  rc],  [co,  re,  fee],  [fer]  ] )  ] , 

in  which  cw  has  been  isolated  as  the  most  connected  component,  and  the  three  subgroups  [tg,  ss,  rc], 
[co,  re,  fee],  and  [fer ]  have  been  collected  for  positioning  around  it.  Accordingly,  the  layout  plan  is 

[{[M3,  []),  <  [[tg,  ss,  rc],  [ co,  re,  fee ],  [fer]],  [cm])] . 

Here  each  tuple  {a,  b)  indicates  a  collection  of  items  (a)  to  be  laid  out  with  respect  to  one  or  more 
other  items  (6).  The  central  component  ([cm])  may  be  placed  arbitrarily,  that  is,  laid  out  with 
respect  to  nothing  ([]).  Figure  16  shows  one  of  the  layouts  synthesized  from  this  plan;  dashed 
outlines  again  show  the  bounding  boxes  for  the  subgroups. 

All  the  alignments  in  Figure  16  are  center-based  (on  the  cm).  In  this  layout,  the  directions 
[Left,  Right,  Above]  have  been  chosen  for  the  respective  subgroups.  When  the  right-hand  element 
of  a  layout  plan  tuple  is  a  collection  of  components  that  lie  within  different  bounding  boxes,  a  ghost 
bounding  box  is  automatically  constructed  to  enclose  them.  This  box  can  then  be  used  to  fix  a 
relative  position  for  the  left-hand  collection(s)  of  the  tuple. 

The  two  subgroup  triples  have  each  been  laid  out  orthogonally  to  the  offset-vector  for  the 
bounding  box  that  contains  them.  This  method  is  standard  for  both  layout  strategies. 


Breadth-First  “Linear”  Strategy 

The  breadth-first  strategy  essentially  tries  to  lay  out  things  in  a  straight  line.  Relaxation  of 
positioning  constraints  to  generate  additional  layouts  permits  a  number  of  exceptions  to  this;  in 
general,  the  strategy  prefers  to  move  from  left  to  right  (or  top  to  bottom),  placing  components 
and  subgroups  as  it  goes.  In  this  strategy,  the  sorting  predicates  are  reversed  from  those  described 
above;  the  layout  starts  with  the  least  connected  item(s)  and  progresses  to  the  more  connected 
ones. 

For  example,  given  the  input  relation 

{  {m,x},  {a,  m},  {/,m},  {m,r,f,c},  { l,d,x,c }  ,} 
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Figure  16.  Center-Out  Layout  Example. 


with  text  labels 


/  lprf", 

/  FCC", 

w  Waveguide", 

d  DSP", 

x  Xmtr\ 

r  Radar  Panel  , 

m  MUX", 

n  . 

a  —*  Antenna  , 

c  — <■”  FCR  Computer" 

representing  a  radar  signal  processing  subsystem,  the  initial  input  transformations  produce  the 
form 

[  (  [<*],  [M]  )>  (  M,  [[/],  [x]] ),  { [/,  x],  [[d,  c}] ),  ( [c],  [[/,  r,  m]] )  ] , 
in  which  the  a  component  has  been  selected  as  a  starting  point.  The  layout  plan  is  then 

[  ( [[«]],  []  >,  <  [M],  MM  [W,  [*]],  H >,  ( [[*,  c}},  [/,  x]  ),  ( [[/,  r,  m]],  [c] )  ] . 

The  meaning  of  the  layout  tuples  is  as  above,  and  the  strategy  again  works  from  left  to  right 
through  the  plan.  Figure  17  illustrates  one  of  the  resulting  layouts. 

Here  the  bounding  box  containing  the  d  and  c  components  has  been  laid  out  with  respect  to 
a  ghost  box  surrounding  the  l  and  x  components,  as  specified  by  the  ( [[d,  c]],  [/,x])  tuple  shown 
in  the  plan.  Since  /  and  x  have  been  placed  at  the  far  ends  of  their  bounding  box,  this  ghost  box 
turns  out  to  be  the  same  as  their  bounding  box. 


Link  Construction 


Note  that  in  the  above  diagrams,  several  links  have  been  joined  to  indicate  common  connect  ions. 
When  an  item  A  is  linked  to  each  of  a  collection  of  items  B\, . . Bn,  and  each  of  B:  and  Bk 
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Figure  17.  Breadth-First  Layout  Example. 


1  <  j,k  <  n  are  linked  to  each  other  as  well,  the  system  will  generally  place  all  the  B ,  in  a 
bounding  box  BB  and  create  a  high-level  link  object  from  .4  to  BB.  A  nexus  point  will  then  be 
created  and  eventually  positioned  somewhere  between  A  and  BB  as  a  target  for  n  +  1  simple 
links — one  from  A  and  n  from  the  J9,. 

In  drawing  simple  links  the  prototype  system  first  attempts  to  use  a  straight  horizontal  or  ver¬ 
tical  line.  If  that  will  not  work  (because  of  interference  from  component  boxes  or  other  already  laid 
out  links),  it  then  attempts  to  construct  a  suitable  link  by  joining  a  horizontal  and  vertical  segment 
as  an  elbow  link.  Figure  18  shows  the  elbow  links  drawn  to  a  “Signal  Processing”  component  added 
to  the  diagram  of  Figure  16. 

The  present  restriction  to  either  orthogonal  lines  or  single  elbows  means  there  are  relatively 
simple  layouts  and  interconnections  that  cannot  be  drawn  entirely  with  “approved"  links.  For 
instance,  removing  the  new  “Signal  Processing”  component  and  instead  connecting  the  “FCC”  and 
“Radar  Control  Panel”  directly  to  each  other  presents  such  a  case.  In  this  instance,  we  would 
probably  like  to  find  a  point  in  the  neighborhood  of  the  added  component  and  use  it.  like  the  nexis 
points  discussed  above,  as  an  intermediate  target  for  the  two  elbows. 
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V.  FURTHER  WORK 


The  work  on  the  table  generator  is  now  fairly  mature  in  that  it  can  generate  at  least  the  basic 
shape  of  all  the  different  kinds  of  tables  we  have  seen.  The  diagram  generator  is  less  mature:  we 
need  to  experiment  both  with  additional  high-level  structuring  strategies  and  layout  strategies  in 
order  to  generate  additional  interesting  displays.  Both  subsystems  require  the  addition  of  a  mor^ 
sophisticated  selector  theory  and  better  customization  cr  formatting  details  as  discussed  below. 


Performance  Measures 


In  order  to  reduce  the  number  of  plausible  displays,  we  want  to  develop  a  measure  based  on  the 
structural  properties  of  the  data  that  could  indicate  when  a  synthesis  branch  would  not  be  worth 
pursuing.  The  measure  will  be  concerned  with  properties  such  as  the  size  of  domain  or  range  of 
relations.  Is  the  mapping  one-to-one  or  almost  one-to-one?  How  many  duplicate  values  are  there 
among  domain  or  range  sets?  For  example,  if  many  values  were  repeated,  the  synthesis  branch  that 
kept  each  repetition  might  not  be  worth  exploring  because  of  the  amount  of  space  each  display 
would  require.  Scrolling  of  displays  offers  another  option  for  extending  the  usable  space  that  may 
or  may  not  be  appropriate  within  a  given  situation. 
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Usage  Measures 


Another  metric  that  could  help  select  among  possible  displays  is  a  model  of  the  amount  of  work 
it  takes  a  user  to  answer  a  given  question  using  a  certain  table.  What  actions  are  required,  such  as 
reading  across  a  row,  looking  up  something  in  a  legend,  or  comparing  two  values?  Can  this  work 
be  quantified? 

A  similar  metric  could  apply  to  the  data  structure  itself  before  rendering  decisions.  The  issues 
would  be  how  many  levels  deep  in  the  data  structure  one  needed  to  look,  what  the  average  or  worse 
case  access  times  would  be,  and  so  on. 

Usage  maximization  becomes  a  multiple  objective  optimization  problem  when  more  than  one 
question  is  to  be  answered  from  the  same  table.  When  does  it  become  beneficial  to  duplicate  t he 
data,  perhaps  in  a  variety  of  tables  each  suited  for  a  particular  need?  When  does  the  cost  of  the 
redundancy  outweigh  the  convenience? 


Variety  of  Displays 

A  natural  extension  of  this  theory  is  to  generate  more  styles  of  displays.  The  work  by  Beach 
(Beach,  1985)  on  composing  tables  could  be  included  in  our  rendering  level.  His  description  lan¬ 
guage  for  tabular  displays  includes  concepts  of  fixed  or  unequal  width  columns,  headings,  under¬ 
lining,  centering,  or  right  or  left  justification,  etc. 

Examples  that  could  be  generated  by  additional  rendering  capabilities  and  data  structure 
reformulations  are  indexes  of  technical  books  (using  a  variety  of  fonts  and  underlining  to  indicate 
the  semantics  of  the  page  reference),  prettyprinting,  and  tree  or  graph  displays. 


Human  Factors 


A  quality  display  synthesizer  should  use  information  about  human  perceptual  limits  to  prune 
the  space  of  possible  displays,  and  later  to  chooae  certain  presentations.  For  example,  the  human 
factors  model  should  know  what  adjacent  colors  or  shadings  can  be  distinguished  by  a  human 
viewer.  Guidelines  about  the  amount  of  white  space  needed  to  enhance  readabilitv  would  be  useful 
in  determining  row  spacing,  column  widths,  and  the  amount  of  text  to  include  on  one  page. 

J.  Mackinlay  (1986)  summarizes  the  effectiveness  of  different  visual  characteristics  such  as 
position,  length,  texture,  area,  density,  shape,  volume,  color  hue,  and  so  on.  with  respect  to  the 
perception  of  three  classes  of  information.  The  three  classes  are  (a)  quantitative  information  (values 
chosen  from  some  range),  (b)  ordinal  information  (ordered  values  but  not  necessarily  numeric,  as 
in  Monday,  Tuesday,...),  and  (c)  nominal  information  (an  unordered  collection  as  in  Argentina, 
Peru,  Brazil,...).  For  any  of  the  three  classes,  people  most  accurately  perceive  a  position-based 
encoding.  People  are  not  good  at  comparing  areas:  using  circles  of  different  diameters  works 
acceptably  to  encode  a  range  of  values  but  it  >s  difficult  for  people  to  perceive  the  encoding  foi 
ordinal  information  unless  the  step  size  between  the  areas  is  great  enough.  And  when  used  for 
encoding  nominal  information,  the  difference  in  areas  will  cause  people  to  perceive  an  ordering  on 
the  items  that  is  not  meant. 
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E.  R.  Tufte  (1983)  similarly  disputes  the  use  of  area  as  an  effective  representation.  He  cites 
studies  showing  how  poorly  people  compare  areas.  For  example,  people  perform  poorly  when  given 
an  initial  circle,  and  when  asked  to  choose  from  other  circles  the  one  containing  twice  the  area  of 
the  given  circle.  This  human  factors  knowledge  would  be  useful  to  make  the  display  synthesizer 
refrain  from  encoding  the  quantity  to  compared  as  area. 

A  notion  of  intent  could  also  guide  the  visual  representation  choices.  Excepting  mandated 
forms,  such  as  accounting  statements,  the  user  wants  to  tell  or  see  a  story  in  the  data. 


Domain  Specific  Graphics 

Each  of  the  major  professions  has  its  own  language,  which  includes  graphical  elements  as  well 
as  words.  Within  a  professional  area,  certain  applications  may  have  their  own  visual  vocabulary. 
Some  visual  conventions  are  specific  to  certain  domains — for  example  the  use  of  red  and  black 
for  loss  or  gain  in  accounting,  and  replacing  trailing  “000”  in  annual  reports  by  one  statement, 
“amounts  are  in  thousands  of  dollars.” 

The  domain-specific  knowledge  should  customize  the  display  synthesizer.  The  conventions  of 
a  field  might  not  be  used  for  a  given  display  but  care  should  be  taken  not  to  violate  them  either. 
It  would  mislead  a  reader  of  a  financial  display  if  the  color  red  represented  credit  entries. 

For  some  domains,  standard  presentations  of  information  are  used.  In  the  project  management 
area,  some  standard  forms  are  PERT  charts,  calendars,  and  GANTT  charts.  MacProject  has  seven 
types  of  charts  derivable  from  planning  information.  Our  synthesis  techniques  should  be  able  to 
synthesize  these  charts.  However,  after  that  capability  is  demonstrated  it  will  be  more  efficient  to 
save  the  display  routine  rather  than  recreate  it  each  time. 

Other  professional  areas  with  their  own  visual  languages  include  computer-aided  design/com¬ 
puter-aided  manufacturing,  program  development  (such  as  SADT  or  Jackson  methods),  decision 
support  systems,  decision  trees,  architecture,  and  each  of  the  physical  engineering  fields. 


Multiple  Displays,  Pages 

Placement  of  figures  on  a  page  or  multiple  pages  needs  to  be  considered  when  displaying  more 
than  one  relation.  If  the  relations  are  analogous,  (e.g.,  showing  the  same  functional  information  at 
different  points  in  time),  then  the  same  style  of  presentation  should  be  used  for  each.  The  synthesis 
process  must  somehow  communicate  among  or  coordinate  these  displays. 

If  a  large  relation  requires  more  than  a  one-page  display,  then  some  information  should  be 
repeated  on  each  page  in  order  to  make  the  display  locally  (to  the  page)  comprehensible.  For 
example,  the  row  and  column  headings  of  a  large  chart  are  often  duplicated  on  each  page.  We  need 
to  decide  what  information  should  be  duplicated  and  how  to  position  it  in  order  for  the  reader  to 
grasp  the  continuity  of  the  relation. 

Thus,  we  will  need  to  explore  the  requirement  for  an  abstract  constraint  solver  that  resolves 
symbolic  descriptions  of  the  visual  display  properties.  For  example,  the  constraints  must  express  the 
notions  of  adjacency  and  overlapping.  Another  type  of  constraint  restricts  the  visual  representation 
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choices.  An  existing  example  is  the  rule  alternating  between  vertical  and  horizontal  alignment. 

In  general,  making  a  visual  representation  choice  at  one  place  in  the  data  structure  might  rule 
out  some  other  visual  choices  elsewhere  in  the  data  structure. 

Our  transformational  capabilities  might  be  able  to  help  fine-tune  the  synthesized  displays  for 
specific  needs.  For  example,  how  can  a  slightly  oversized  display  be  transformed  to  fit  within  a 
page?  What  information  do  we  choose  to  omit,  abbreviate,  move,  or  shrink?  What  transformations 
could  be  used  to  help  a  display  that  doesn’t  work? 

In  summary,  we  are  solving  design  problems.  Constraint-based  layout  of  displays  has  the  same 
challenges  as  designing  offices,  work  areas,  engines,  electrical  wiring,  or  circuit  board  layouts.  The 
techniques  being  developed  for  constraint-based  programming  will  contribute  to  the  design  problem 
definition  and  solution. 


Interactive  Graphics 

There  is  significant  literature  in  the  proceedings  of  the  Association  of  Computing  Machinery 
(ACM)  annual  SIGGRAPH  conference  and  the  ACM  Transactions  on  Graphics  journal  entitled 
User  Interface  Management  Systems  (Olsen,  1987).  Areas  of  active  research  include  specification 
languages  for  designing  interactive  displays,  and  toolboxes  for  implementing  them. 

User  interaction  during  the  synthesis  of  the  graphics  display  could  help  reduce  the  search  space 
by  proceeding  more  directly  to  the  style  of  display  a  user  needs.  Possible  directives  that  could 
be  given  by  the  user  as  she  or  he  views  a  partially  synthesized  display  include  instructions  to 
swap  axes,  reorder  columns,  move  labels,  or  choose  an  undo  option.  However,  we  doubt  there  is  a 
natural  direct  manipulation  interface  that  would  achieve  the  deep  restructuring  performed  by  our 
reformulation  steps.  That  means  the  user  may  be  able  to  direct  the  search  within  a  certain  branch 
of  the  tree  but  not  jump  across  branches. 


Related  Work 


The  work  most  similar  to  ours  is  that  of  Mackinlay  (1986)  whose  viewpoint  we  share  to  a 
large  extent.  His  work  explores  the  use  of  graphical  languages  for  encoding  relational  information. 
The  use  of  visual  associations  used  in  our  paper  can  be  considered  to  be  an  instance  of  this  idea, 
specialized  for  our  particular  applications.  Only  a  few  of  the  kinds  of  tables  and  diagrams  presented 
in  our  paper  are  considered  by  Mackinlay:  offset  plots,  and  diagrams  with  very  simple  layout. 
Another  difference  is  Mackinlay’s  emphasis  on  effectiveness  of  displays  (our  “selector”)  as  opposed 
to  our  emphasis  on  a  fine-grained  generator  of  displays. 

Several  researchers  have  developed  automated  layout  systems  for  tree-structured  diagrams, 
(Wetherell  &  Shannon,  1979),  (Reingold  &  Tilford,  1981),  (Robins,  1987),  and  (Moser,  1990).  Rel¬ 
atively  little  work  has  been  done  automating  the  layout  of  general  graph  structures.  An  algorithm 
for  laying  out  Entity/Relationship  diagrams  is  described  (Tamassia,  et.  al.,  1983).  It  uses  a  fixed 
reference  grid  for  positioning  entities,  performs  several  graph-theoretic  transformations,  and  solves 
a  linear  programming  problem  in  attempting  to  minimize  edge  crossings  and  total  edge  path  length. 
Their  layouts  are  more  restricted  than  ours  and  do  not  include  any  subgrouping  mechanisms.  We 
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have  not  as  yet  applied  any  formal  minimization  techniques,  such  as  linear  programming,  to  our 
link  construction  methods. 

Tichy  and  Newbery  (1987)  discuss  a  knowledge-based  graphical  editor  with  capabilities  similar 
to  ours  for  adding/deleting  components  and  edges,  highlighting  or  shading  individual  components, 
and  so  on.  They  also  concentrate  on  minimization  of  edge  crossings  and  are  investigating  mecha¬ 
nisms  for  accommodating  user  specification  of  layout  style,  such  as  component  grouping,  bounding 
box  abstractions,  and  relative  positioning.  Our  system  already  uses  these  mechanisms;  we  are  also 
looking  into  the  construction  of  multilevel  diagrams  and  facilities  for  zooming  in  and  out  between 
the  levels. 

There  are  many  commercial  word  processing,  spreadsheet,  and  database  programs  with  table- 
displaying  components.  We  know  of  none  concerned  with  the  kinds  of  restructuring  of  displays  as 
described  in  this  paper. 

The  October  1989  IEEE  Computer  Special  Issue  on  Visualization  in  Computing  (IEEE  Com¬ 
puter,  1989)  describes  recent  work  on  generating  visualizations  for  use  in  computing  environments. 


VI.  CONCLUSIONS 


We  have  developed  a  generative  theory  for  the  automated  synthesis  of  tabular  and  diagrammatic 
displays.  The  prototype  based  on  this  theory  demonstrates  that  a  small  number  of  design  principles 
can  succeed  in  creating  a  variety  of  interesting  and  useful  displays.  The  success  of  this  approach 
raises  hopes  for  increasing  the  range  of  problems  tackled  by  automated  programming  to  include 
useful  and  creative  input/output  capabilities.  Continuation  of  this  effort  could  help  reduce  costs  of 
porting  programs  to  new  hardware  by  replacing  handcoding  with  resynthesis  of  the  display  code. 

The  theory  can  be  extended.  New  data  structure  reformulations  and  new  visual  display  pos¬ 
sibilities  can  be  added  to  the  existing  framework.  Application-specific  display  knowledge  could  be 
added.  A  theory  for  the  selector  component,  and  performance  and  usage  measures  will  complement 
the  generator. 

In  summary,  parallels  exist  between  display  synthesis  and  program  synthesis  that  have  not  yet 
been  mined  by  our  research  effort;  the  path  begun  here  seems  to  lie  in  a  rich  vein. 
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